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(57) Abstract 

An apparatus and method of altering video images to 
enable the addition of images, message, slogans or indicia in 
such a manner that they appear to be part of the original image 
as displayed. The application of the apparatus and method is 
particularly adapted to be used as advertising during sporting 
events or other events whereby an operator selects a portion of 
the video image display (16) such as a portion of a tennis court 
(12), recognizes the selected portion (17) and inserts a logo or 
advertising message (27) into that selected portion. The mes- 
sage is inserted into the selected portion of the court independ- 
ent of how the scene is being panned or viewed by the televi- 
sion camera, The apparatus employs a pattern recognition al- 
gorithm such as the Burt Pyramid Algorithm to recognize the 
specific selected image portion. 
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Television Displays Having Selected Inserted Infljpja 

Related Applications & p riority DQaomentg 
5 This invention is related to British Patent 

Application Serial No. 9102995* 9 , entitled Electronic 
Billboard: A Method of Advertising Using Existing 
Television Transmission Facilities, filed February 13 , 
1991 by Roy J. Rosser & Martin Leach, which 

10 application is based upon a provisional application 
filed February 14, 1990* In addition, this 

application is related to British patent Application 
Serial No* 9019770*8, entitled Audio Enhanced 
Electronic Billboard, filed September 10, 199 o by Roy 

15 J. Rosser. 



Technical Field 

The present invention relates to a method of 
advertising using conventional television transmission 
20 facilities. In particular, the invention relates to 
methods for merging or inserting a chosen image into a 
preselected existing image portion of a television 
service. 

25 Background Arfc 

The conventional methods for merging video 
images include the "blue screen" or "chromo-key" 
techniques. For example, these techniques are used in 
the broadcast industry to show weather forecasters 

3 0 positioned in front of weather maps • In this 
technology, the forecaster is filmed in front of a 
screen having a particular pre-determined color- The 
screen color is usually blue, hence, the name of this 
technique. The technique calls for substituting the 
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second image (the weather map) at all points in the 
first image (the forecaster) where the signal contains 
the blue color. The result is that the image of the 
map and forecaster will appear to merge. However, 
5 thi s method has two short comings. First, the 
forecaster must not have any item on his person that 
is the same color as the blue color of the screen. If 
he does, and the second image is merged with the first 
image, any item on the forecaster having the 

10 predetermined blue color will appear as a hole, 
second, this technique does not provide for the 
integration of the two images. That ie, the two 
images are merely superimposed. That is why the 
forecaster must make sweeping and general motions to 

X5 indicate positions on the map. This limitation is 
important when considering this technique for an 
advertising application. 

There are of course other techniques which 
enable the insertion of video images into an existing 

20 image which are basically employed at the studio and 
which utilize complicated console devices to 
superimpose a video picture on another video picture. 
Essentially the present technique involves a 
completely different approach with great applicability 

25 to television advertising. As will be explained, the 
present application relates to a method of advertising 
using existing television transmission facilities in 
which the advertiser selects predetermined areas in a 
video scene. These areas are then recognized using 

30 existing technology as pattern recognition techniques 
and the content of these areas is replaced by 
inserting an image or images of the advertisers 
choosing. In order to accomplish the present 
invention one must recognize the selected 
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predetermined area and replace the same with 
advertising indicia in real time. The inserted 
indicia is then blended into the original video image 
in such a way that it appears part of the television 
5 scene and cannot be otherwise discerned by a typical 
viewer* The inserted advertising or messages can 
further be highlighted to the viewer by moving the 
message, changing color, associating the message with 
sound and otherwise imparting modulation to the 

10 message to distinguish the message during the course 
of the television program. Such techniques will be 
further described. Basically, it is believed that the 
techniques to be described herein have extreme 
usefulness and applicability for the display of 

15 advertising in television sporting events such as 
tennis matches, boxing, baseball, football and so on. 
While there is particular applicability to such events 
the techniques described may be adapted to all formats 
of video presentation and have general applicability 

20 to the field of advertising. 

As will be described and in order to 
accomplish this result, one must basically select an 
object or an area within a given video image. For 
example the area selected may be a televised tennis 

25 court. This area is then recognized in each of a 
subsequent stream of video images and once recognized 
the video content of that image is replaced with a 
desired content which may be an advertising 
commercial, logo or some other matter* The insertion 

30 and replacement of the existing video image with the 
new material is accomplished completely independent of 
the size of the image in subsequent television scenes 
or frames. The system described herein will always 
recognize the particular area selected no matter what 
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30 



the size of the area Is with respect to the remaxning 
television picture. This, therefore, creates ^ the 
illusion to the viewer that the replaced sublet 
matter is actually that subject matter which xs being 
televised in real time. Thus, the methods ana 
apparatus described enable advertisers to add or merge 
J^ges, message, or slogans to preexisting video 
images in such a way that they appear to be part of 
the original image even when the original video stream 
is of a live event being distributed in real time. 

Thus, the techniques described herein are 
superior to prior art techniques of inserting video 
information. Xn order to do and accomplish the 
results of this particular application an electronic 
-aevice-is-reguired to select, recognize and substitute 



images to be broadcast on live televised events. The 
device detects part or all of an object or objects 
within a video image and uses the position of the 
object or part thereof as a reference. The devxee 
20 then positions another still or video image into the 
original image at the reference location. The stxll 
or video image is positioned accurately so that the 
final composite image appears as though it xs part of 
the original scene. That is, the added image is 
seamlessly and realistically incorporated into the 
original event. The appearance of the added xmage 
will conform to the appearance of the original scene 
when the scene is moved, panned, magnified, zoomed or 
otherwise altered in size or perspective. The System 
can also compensate for less distortions by mappxng 
the distortions in a televised scene and using the 
mapped distortions in future processing. All this 
must be done in real time, or with a processing delay 
being sufficiently small that it can be applied to 
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events being transmitted live. The -Burt Pyramid- « 
the most appropriate way to do this amount of 
computation on a video image in such a short time wath 
existing technology. This is a veil know method of 
processing, analyzing and/or synthesizing signals and 
is described in TJ.S. Patent No. 4,674,125, entitled 
Real-Time Hierarchy Pyramid Signal Processing 
Apparatus by C.R. Carlson, et al. and issued on June 
16, 1987. The Burt Pyramid is also discussed in the 
publications entitled -Past Algorithms for Estimating 
Local image Properties-, by Peter J. Burt, Computer 
Vision, Graphics and Image Processing 21, pp. 368- 
382, 1983 and -Pyramid-Based Extraction of Local Image 
Features with Application to Motion and Texture 
15 Analysis-, by Peter J. Burt, SP1E, Vol. 360, pp. H4- 
124. 

The "chromo-Xey" or -blue-screen- technology 
may be used in certain applications as an adjunct to 
the pattern recognition method. This technology is 
20 one way of distinguishing non-replaceable, foreground 
pixels from replaceable, . background pixels. For 
instance, in a sport such as tennis, it is usually 
necessary to ensure that only the tennis court is 
replaced with a message merged onto the field of 
25 play. The players or their equipment should not toe 
affected. one way to accomplish this is to replace 
only the green colored pixels that are in the merged 
image and are within the color range of the court. 
Alternate methods of distinguishing between 
replaceable and non-replaceable pixels include storing 
an image of the unoccupied court and subsequently 
using that image as a comparison to determine which is 
the player/equipment and which is the background. 
Another method uses differential motion between the 
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objects or markings on the court and the 
equipment/Players moving over the court to 
distinguish between replaceable and non-replaceable 
pixels . 

5 Although the pattern recognition software 

using the "Burt Pyramid" algorithms is fast, it still 
takes a finite time to operate. Therefore, it may be 
necessary to introduce a time delay into the system. 
The delay may be one or two frames, or it may be 
10 longer. A frame store or other similar means can be 
used to temporarily locate a small number of the video 
screens or frames. Incorporating a time delay unit 
into the present invention will be discussed 

hereinafter. 

_other conventional methods of processing 
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video signals and images or the like are presented 
below. Some of the disclosed methods also use the 

Burt Pyramid. 

An example of an apparatus that analyzes 

20 signals and the frequency spectrum for the frequency 
components of the signals is disclosed in U.S. Patent 
No. 3,731,188, Signal Analysis of Multiplieatively 
Related Frequency Components in a Complex signal, 
which issued May 1, 1973 to S.E. Smith. Other patents 

25 disclosing subject matter related thereto are U.S. 

Patent Nos. 3,140,710, 3,315,222, 3,470,468, 3,473,121 

and 3,582,957. 

An example of an image processing method for 
reducing noise in the image is disclosed in U.S. 
30 Patent No. 4,442,454, Image Processing Method Using a 
Block Overlap Transformation Procedure, which issued 
April 10, 1S84 to P.G. Powell. In the method 
disclosed, the image is divided into blocks of image 
elements which are transformed according to the linear 
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transform procedure. Other patents disclosing subject 
matter related thereto are U.S. Patent Nos. 3,715,477, 
3,996,421, 4,055,756, 4,057,828, 4,134,134, 4,205,341, 
4,242,705 and 4,245,248. 
5 An example of a method and apparatus for 

transforming numerical signal data of a video signal 
is disclosed in U.S. Patent 4,447,886, Triangle and 
Pyramid Signal Transforms and Apparatus, which Issued 
May 8, 1984 to G-W. MeeXer. As disclosed in this 
10 patent, the numerical signal data of the video signal 
is subjected to a transform domain where the signal is 
processed and reconstructed into the original domain 
to provide a replica of the original data. Other 
patents disclosing subject matter related thereto are 
15 U.S. Patent Nos. 2,921,124, 3,051,778, 3,742,201, 
4,222,076, 4,261,018, 4,261,043 and 4,286,330. 

An example of an image-processing system 
employing coring techniques to reduce a noise 
component of an image signal is disclosed in U.S. 
20 Patent No. 4,523,230, System for Coring an Image- 
Representing signal, which issued June 11, 1985 to 
C.R. Carlson, et al. The Burt Pyramid spectrum 
analyzer and synthesizer (U.S. Patent No. 4,674,125) 
may be employed with the system disclosed in the '230 
25 patent, so that the system can operate in real time on 
a video signal of a scanned television image. Other 
patents disclosing subject matter related thereto are 
U.S. Patent Nos. 4,163,258 and 4,463,381. 

An example of a method and an apparatus for 
30 reducing image data while retaining the ability to 
observe objects with high spatial resolution is 
disclosed in U.S. Patent No. 4,692,806, Image-Data 
Reduction Technique, which issued September 8, 1987 to 
C.R. Carlson, et al. Other patents disclosing subject 
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matter related thereto are U.S. Patent Nos. 3,016,518, 
3,983,328, 4,405,943 and 4,513,317. 

An example of an image processing method 
for filling in one or more void defects in an iaaage to 
5 provide a natural looking processed image is disclosed 
in U.S. Patent No. 4,698,843, Method for Compensating 
for Void-Defects in Images, Which issued October- 6, 
1987 to P.J. Burt, et al. This invention discloses 
employing Ibe Burt Pyramid as disclosed in U.S. Patent 
10 No. 4,674,125. Other patents disclosing subject 
matter related thereto are 0.S- Patent Nos. 3,737,855, 
3,887,762, 3,973,239, 4.000,399, 4,010,446 and 

4,661,986. . 

THus as one will ascertain the advertising 

15 method and apparatus of this invention requires very 
precise positioning of a new image into an existing 
image. The techniques to he described employ pattern 
recognition apparatus which recognizes pre-selected 
features or portions of a typical television scene. 

20 These features can then be used to locate the 
position, size and perspective of an artificial 
electronic advertising message which Is added in the 
exact congruency with the replaced recognized image 
and appears to the end user as if it were part of the 

25 original scene. The added material is inserted into 
the pre-selected scene independent of size of the 
selected feature as varying on a frame to frame basis. 
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nisclosure of t hfl invention 

Apparatus for altering a video image display 
to provide a substituted display of desired indicia 
within a preselected portion of said video image 
display and independent of the size of said selected 
portion which siae is strictly a function of the TV 
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camera, said TV camera operable at various different 
perspectives employed to create said display, 
comprising means for selecting said portion of said 
display to be substituted , means responsive to said 
5 video signal display for recognizing said selected 
portion of said display, means for generating video 
signals indicative of said desired indicia to be 
substituted, means responsive to said desired indicia 
signals and said video signal for inserting said 

10 indicia into said video image display at said 
preselected portion. 
Brief Description of the Figures 

Fig, 1 is a detailed blocked diagram showing 
an apparatus for recognizing video image formats and 

15 Inserting desired indicia therefore according to this 
invention. 

Figs. 2, 3 and 4 are different perspective 
plan views of a television image of a tennis court 
showing how the logo is positioned by the system of 
20 Fig. 1 on the tennis court as a function of different 
perspective views of the court as taken by a typical 
television camera. 

Best Mode Fo ir Carrying Out the Invention 
25 Referring to Fig, l f there is shown a block 

diagram of a general format of a system employing the 
apparatus to be described. . Before proceeding with a 
description of the apparatus depicted in Fig. 1, the 
following comments are pertinent. 
30 The uses of the apparatus and method relate 

to advertising and include but are not limited to 
advertising which occurs during sporting events where 
a field or other scene is normally depicted such as a 
tennis court, a baseball field, a football field and 



8NS0OCIEX <WO 9302524A1 



WO 93/02524 



PCT/US91/05174 



-10- 



10 



so on. In such arenas or stadiums there is normally 
advertising present on the fences or elsewhere. This 
advertising can continuously be recognized toy the 
techniques to be described, and the advertising which 
is permanently painted on a fence or displayed on a 
fence can be replaced. In a similar manner certain 
areas of the playing field which do not have 
advertising can now include advertising such as 
advertising appearing on a tennis court or playing 
fields. By making the advertising a discrete part of 
the action the game does not have to be interrupted. 
Even in sports where there are breaks for advertising 
the method and apparatus to toe described is of benefit 
to advertisers because the message can be displayed 
,hi-ie— the— audienceJs— attention is focused on the 
action. Apart from this particular application, which 
relates to advertising, one can also recognize by 
means of the pattern recognition schemes and boundary 
lines which are associated with a typical sports court 
such as the serve line in tennis, the foul line in 
basketball, the various field lines in baseball, 
football and so on. These lines can be further 
recognized and enhanced by artificially including 
markings for the various field or other lines or 
25 boundaries. Apart from such other items, advertising 
messages can be displayed and then changed and so on 
during a single televised event. 

Referring to rig. 1, there is shown a 
replica of a typical tennis court 12. The tennis 
court is separated by a net and has a back court 
designated 11 with a fore court 13. The tennis 
court 12 is shown in a perspective view. Shown is a 
television camera 14 which is in the process of 
recording the sporting event or tennis match. The 
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opponents are not shown but would be included in a 
typical tennis match. As will be explained in one 
embodiment of the invention, the fore court 11 as well 
as the back court 13 or the entire tennis court 12 is 
5 subjected to a pattern recognition algorithm whereby 
an advertisement is actually inserted directly on the 
portions of the court as shown in Fig. 2. 

As shown in Fig. 3, the same advertisement 
or logo is depicted independent of the perspective 
10 view of the court and independent of whether the 
television camera 14 is employing a zoom lens or the 
scene is televised by an additional camera coupled to 
suitable equipment. The cameras provide a view of the 
tennis match through a different perspective or 
IS different visual planes. Thus as one can ascertain 
from Figs. 2, 3 and 4, the logo will appear in exact 
conformity with the perspective view of the court so 
that it will appear as if the logo were permanently 
imprinted on the actual court being televised. Thus, 
20 the television camera 14 is shown televising the 
tennis match. As will be explained, the output of the 
camera 14 is coupled directly to a main image 
television receiver 16 which receives the video signal 
from the camera and which provides a display of the 
25 television signal as shown on the display screen. The 
camera 14 is also coupled to one input of an image 

analyzer 17. 

The equipment also includes a microphone 15 
utilized to pick up sound from the area being 
30 televised with the output of the microphone coupled to 
an input of the image analyzer 17 and also coupled to 
the input of an audio analyzer processor module 26. 
The output from the audio analyzer processing module 
26 is directed to an input of a second image source 
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27. The second image source, as will be explained, 
operates to provide the advertising message or indicia 
which may be a slogan, an image, a logo and which 
advertising message will be displayed on the selected 
portions 11 4 13 of the tennis court as shown in Figs. 
2, 3 and 4 and designated as logo. The second image 
source 27 has an output coupled to an Image 
synthesizer 20. The image synthesizer has an output 
coupled to the image analyzer 17. Both the image 
synthesizer and the image analyzer have inputs and 
outputs directed to a frame storage and delay device 
19 which will he subsequently explained. Also seen is 
that the image synthesizer has one output coupled to 
one terminal of a switch 32 which interfaces with a 

15 transmission/reception system 31 . 

The other terminal of the switch 32 is 
coupled to a television receiver 30 which as seen in 
Pig. 1 has the inserted logo positioned on the 
selected portions of the tennis court. The television 
receiver 30 is indicative of a user's television as 
for example used by a consumer viewing the sports 
event. The operation of the system is basically as 
follows. Before proceeding further it is understood 
that the camera 14 and microphone 15 in conjunction 
25 with the above noted equipment can directly operate to 
provide a normal television signal to be transmitted 
by the transmission and reception system 31 to remote 
television receivers as 30 as is conventional. Hence, 
it is indicated that the television camera 14 as well 
as the microphone 15 and associated equipment are 
conventional components and therefore capable of 
providing a typical, normal, unaltered television 
signal and display. The distribution of the television 
signal is accomplished via the transmission/reception 
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system 31 which aci indicated may be a wireless 
broadcast or a cable network* Such systems for 
distributing television signals are extremely well 
known. If the sender desires to insert a given 
5 advertising message ior a logo in a selected area of 
the television display, the following operation 
occurs • 

The image of the event as present on the 
screen of the television receiver 16 will eventually 

10 be altered to incoirporate the advertising message 
which as indicated may be a slogan or image. The 
alteration may take place at the originating site of 
the event, at a network broadcast center, a local 
broadcast center, at a local broadcasting station # at 

15 a cable headend or ahy other site where video signals 
are disseminated* As one will ascertain, the 

equipment shown in rig. 1 is universally adaptable to 
be utilized at ahy particular location and is 
relatively small and transportable. The added message 

20 which is provided by the separate video source 27 may 
be prerecorded on a tape or stored by other means and 
eventually will be merged into the video stream by the 
image synthesizer 20L The image synthesizer 20 uses 
data from the image: analyzer 17 and from the frame 

25 storage and delay device 19. The essence of this 
method of advertising is to incorporate the added 
message from the secondary source 27 into the main 
image recorded by camera 14 so that the end viewer who 
is watching the television screen associated with 

30 receiver 30 believies that the image is actual 
imprinted or otherwise permanently located on the 
tennis court at the actual site of the match. 

Also, as shown in Figs. 2, 3 and 4 the logo 
is actually added to j the television court display even 
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when the television camera is turned, panned and or 
zoomed to follow the action of the match progressing 
on the tennis court 12. This merging is seamless and 
incorporates the logo or advertising message into the 
5 video image from the camera 10. The hardware employed 
incorporates software programs which are all known in 
the prior art and examples of which have been given 
above. Essentially, the merging of the advertising 
message or image into the video images is accomplished 
X0 in the image analyzer 17. The image analyzer 17 
operates in conjunction with the image storage and 
delay module 19 as well as the image synthesizer 20 to 
provide the composite picture as shown on the screen 
of television receiver 30. The image analyzer 17, the 

_ 15 image synthesizer 20, the delay module 19 and the 

video source 27 as well as the audio "analyzer and 
processor 26 constitute the modules which operate to 
provide the merging of the advertising message with 
the video signal as will be further described. 
20 The image analyzer 17 includes a 

conventional computer such as a PC or mini-computer 
including a suitable memory such as a hard drive and 
so on. The image analyzer computer 17 is programmed 
to take the television signal emanating from camera 14 
25 to store the signal and to process the stored signal 
according to a selected pattern recognition technique 
so that the incoming video image is processed in real 
time. The image analyzer 17 may be associated with 
suitable operating interface equipment such as a 
30 keyboard, a light pen or a mouse to allow the operator 
to originally select the landmarks desired in the 
original scene. in this manner the operator will for 
example outline the tennis court 12 as present on the 
screen of the monitor receiver 16X. The monitor 
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receiver 16 is associated with a computer interface 
and provides the television image on the screen which 
image can he interfaced hy the system operator. This 
outline will be stored in memory in regard to digital 
5 bits or pixels in the image analyzer ♦ As one can 
ascertain , the entire video signal as displayed on the 
screen is broken up into pixels where each television 
line and each position along a line is represented by 
a suitable digital number corresponding to the color 

10 intensity or content at the line position or to the 
monochrome content of the line position. 

Techniques for converting television signals 
Into digital signals including pixels are extremely 
well known. Hence one can store the entire 

15 representation of a complete television frame or image 
within a conventional memory by conventional 
techniques. The light pen or other marking device 
which is associated with the image analyser provides 
an outline of the image which is to be recognized by 

20 pattern recognition program such as described in U. S. 

Patent 4,674,125 wherein the Burt Pyramid Algorithm is 
employed as one well known technique* While the Burt 
algorithm as indicated above is extremely useful in 
implementing pattern recognition to therefore enable 

25 the system to recognize the tennis court, many other 
techniques for pattern recognition including other 
algorithms may be used. There have been entire books 
and volumes written on pattern recognition and image 
processing. See for example "The Proceedings of the 

30 IEE W , May, 1979 issue entitled "Pattern Recognition 
and Image Processing", Volume 67, Num. 5, Pgs+ 705 * — 
880- Many of the above noted patents describe various 
other techniques for performing pattern recognition. 
The image analyzer 14 as well as the image synthesizer 
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22 are electronic or optical processors and include 
software capable of implementing the Burt pyramid or 
other pattern recognition techniques and for 
processing the incoming video images in real time, 
such algorithms such as the Burt pyramid algorithm 
employ a particular sampling technique analyzing a 
relatively high resolution video signal into a 
hierarchy of N (where M is a plural integer), separate 
component images in which each component image is a 
Laplacian image comprised of a different octave of the 
spatial frequencies of the original image, plus a 
remnant Gaussian image ♦ 

The term "pyramid" relates to the successive 
reduction in spatial frequency and sample density of 

15 each- of the -hierarchy _of the component im ages i n going 

from the highest octave component to the lowest octave 
component image. This algorithm is computationally 
effective and can deal with visual images that are 
intended for viewing by human beings within the 
typical television frame rates. There are other 
techniques which can also be employed to process the 
image, and, as indicated, the frame and storage 
device 19 can for example delay many frames and hence 
the processing can be slowed down a great deal 
25 depending on whether or not one wishes a simultaneous 
transmission of the actual event or a delayed version 
thereof . Operation of the system as indicated is 
basically as follows. The operator views the image as 
presented on the monitor television receiver 16 and 
marks selective portions of the outlines of the court 
or the net area to choose or to direct that portion of 
the displayed image to be recognized and where the 
advertising indicia will be inserted. In this 
particular example the operator will mark the upper 
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30 
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court 11 and the lower court 13 by mean© of a light 
pen or other suitable device and therefore instruct or 
command the image analyzer 17 to recognize these 
markings in subsequent video images of the court 12 as 
5 televised by the camera- The image synthesizer is now 
employed to locate, position and orient , including the 
correct magnification, the indicia or logo, which is 
taken from the second image source 27 so that it 
appears as part of the original scene. The logo wliich 

10 is generated by the second image source 27 is made to 
appear as part of the background and does not 
interfere with objects or people in the foreground. 
This is accomplished by allowing the logo to key over 
specific colors or ranges of colors such as the color 

15 of a court in the tennis match. 

As one understand a tennis court for example 
is a certain shade of green. Thus the areas 11 and 13 
are green or some other color which completely 
differentiates from the colors of the uniforms as well 

20 as the skin of the players or participants. By means 
of conventional RGB color processing the image 
analyzer can therefore store information relating to t 
1) the outline of the court as selected by the 
operator and 2) the color of the court as further 

25 evidenced by the detail in the video image. in this 
manner the image analyzer 17 transfers information to 
the image synthesizer 20. The image synthesizer 
receives information relating to the position and 
location of the court portions 11 and 13 . The image 

30 synthesizer 20 then takes the video information from 
the second image source 27 and inserts this 
information in the images of the court section 11 and 
13. This insert appears as the logo which is depicted 
in Figs. 2, 3 and 4. This insertion is done by means 
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of the frame storage and delay device 19 wherein the 
video images are stored. The information from the 
image synthesizer iB now inserted in absolute proper 
location, position and orientation including the 
5 correct magnification into a stored picture Which 
picture is now re-routed through the image synthesizer 
to the transmission and reception system 31. This 
causes the display on television receiver 30 to be 
shown, as indicated. The distinction between the 

10 foreground and background may also be identified by 
other techniques apart from color detection. It is, 
of course, clear that by not keying over specific 
colors such as clothing and flesh colors one will only 
provide the logo in the areas 11 and 13 and the logo 

15 ^wi-l-l -be-obscured^by^the_playe rs as to only _the portion 
of the player that overlies the logo- Hence it 
appears as if the logo is actually painted directly on 
the tennis court. 

Apart from color recognition, one can 

20 distinguish between the tennis court and the 
background of the court by other techniques such as 
optical image analysis and so on. For example, motion 
regarding the video content can be easily detected by 
responding to frequencies present at the scanning rate 

25 (horizontal). Thus, the prior was capable of 
detecting motion by responding to such frequencies. 
Hence this system-can- be --used- t^-teiscrt th»~logo-only 
in the selected area and only when motion is not 
detected. Motion can also be detected on a frame-to- 

30 frame comparison of video content. The resultant 
effect is to allow a viewer to watch the televised 
match discreetly and completely as it is occurring 
with advertising conspicuously added directly on the 
court. The advertising or logo could be of any 
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particular format such as numbers, objects and so on. 
It is, of course, understood that the pattern 
recognition technique can discriminate and select 
almost any object present in a television picture* 
5 For example, the operator may select the back portion 
of a player's uniform and the pattern recognition 
algorithm will recognize that back portion of the 
player and the logo is inserted by means of the 
second image source 27 on the player «s back. Thus, 

10 apart from a tennis court, one can insert 
advertisements on the back wall of a baseball field or 
on the pitcher" s mound or any other selected place. 
It is actually up to the advertiser and the user to 
determine where the logo is to be inserted, and the 

15 system assures that the logo inserted will always be 
in proper perspective no matter how the -particular 
sport event being televised is viewed. 

The frame storage and delay device 19 
operates to allow the analyzer 17 as well as the image 

20 synthesizer 20 to process the signal in real time. 
Thus, the time delay module receives the signal and, 
basically, is a frame store or a multi-frame store 
device. The frame and storage delay module 19 can 
utilize CCD devices or MOS technology or other frame 

25 storage devices and technologies* There are virtually 
many devices in the prior art which can store single 
frames or multiple frames of video. The frame storage 
device, again as indicated, can be any memory medium 
that has relatively fast read/write times so that it 

30 can compatibly operate with the video signal. The 
amount of delay and storage depends on the speed of 
the analyzer 17 as well as the quality of the image 
from the video camera 14. For larger and high 
definition . video formats one would require more 
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mem ory. The memory can be on an optical disc or can 
be implemented by other well known memory techniques. 

The present system utilizes the microphone 
15 which as indicated is coupled to an audio analyser 
5 processor 26 for accommodating special effects. The 
audio analyzer processor 26 again can be a mini- 
computer or a high speed PC together with a hard disc 
or other type memory. The audio processor 26 as well 
as the microphone 28 are included, and basically the 

10 microphone is a conventional component for providing 
sound to enable the transmission of normal televisxon 
signals. The audio processor 26 monitors the audio 
stream emanating from the microphone 28 and the sound 
as processed by the processor is used to control 

_ XS aspects^pf_the_ima ge as inserted J>y the second image 

video source 27. Thus, the audio processor can 
respond to the magnitude of the signal from the 
microphone 15 and alter the logo which is displayed on 

the court accordingly. 

20 FQr example, let us assume that a point is 

made by one player after a rather spectacular rally. 
In these instances the crowd usually responds by loud 
clapping and cheering. In this way in order to fully 
emphasize the reaction of the crowd one can utilize 

25 the increased sound level to modulate the logo. For 
example, to turn the logo on and off, or to change the 
color and hence add dimensions to the overall 
televised presentation to further enable the 
advertiser to obtain maximum benefit from the 

30 advertisement. such techniques are relatively simple 
to implement and are well within the ordinary ken of 
one skilled in the art. The audio processor can also 
include a voice recognition system which systems are 
commercially available. Thus, particular words 



or 
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statements can be recognized by the audio processor as 
those words or statements spoken by a commentator. In 
this manner the inserted video can be made to respond 
accordingly. 

5 For example, certain baseball commentators 

are famous for their particular expressions such as 
the expression "holy cow" when the commentator uses 
this expression one Knows that either a home run or 
some other spectacular event has occurred* in this 

10 way the indicia or logo is inserted by means of the 
pattern recognition processing technique described 
above and can be modulated or suitably altered in 
regard to such recognized words or phrases. The 
applications which are available by utilizing the 

15 above described technology are too numerous to mention 

these techniques. Thus, the extensive prior art 
listed in the beginning of this application is 
indicative or the various different techniques which 
20 can be employed to implement the above described 
operation. As one can immediately ascertain , the 
potential uses of the methods and apparatus described 
involve advertising, but as indicated, they can be 
utilized in directing the viewer's attention to 
25 particular places on the sports court such as 
highlighting the outlines of the court. 

Another technique which can easily be 
employed by using the techniques is the marking or 
modulation of the progress of a ball being utilized in 
for example a tennis match or during a baseball game. 
As one can ascertain in many scenes which are 
displayed on television, it is extremely difficult to 
follow the flight of a pitched or struck ball. Based 
on the real time operation of the algorithms described 
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for pattern recognition, it would be relatively easy 
to recognize the ball as thrown for example between 
and a pitcher and catcher. In this manner the 
pattern recognition apparatus would recognize the ball 
5 and one can modulate the path of the ball on a frame- 
to- frame basis thus enabling the viewer to clearly 
distinguish the ball and to visualize the ball during 
the course of the baseball game* The ball can also be 
recognized in regard to that employed in a tennis 

10 match. As one can understand, in certain tennis 
matches such as those played on clay* it is often 
difficult to see the ball and hence the above 
techniques can be utilized in enhancing the game by 
recognition of the ball on a frame-to-frame basis, 

15 thereafter marking or inserting indicia indicative of 
the trajectory of the ball* 

The applications of the above technology in 
regard to advertisements are many and include placing 
advertising in and around the court of play, placing 

20 advertising on a bowling ball or a basketball (either 
of which are easy to recognize) placing advertisement 
on the bowling alley lane or parts of a basketball or 
tennis court including the backboard of a basketball 
court or the net of a tennis court. These techniques 

25 will enable one to provide inserted patches on 
individual players such as on the arm or the back of 
the player which then can be replaced with 
advertisements which can be varied during the course 
of a match or can be different in different parts of 

30 the country or different parts of the world- One can 
utilize the inserted video display to provide 
advertising effects such as modulating the display 
when a particular event is achieved. One can also 
utilize the above described techniques to actually 
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insert score boards in regard to certain games. For 
example, a score board could be superimposed on the 
fence in a baseball game or elsewhere where such 
facilities do not exist - 
5 While it is understood that the block 

diagram and specification herein depicted in Fig. 1 
are devoid of any specific examples of algorithms or 
of the programs utilized to implement the technology/ 
it is again indicated that the pattern recognition and 

10 algorithms and techniques are all well known in the 
art, and any one of such algorithms can be 
successfully employed and incorporated in this system 
with a minimum of effort. Those listed in the many 
patents and articles and other presentations cited in 

15 this application are applicable. It is, of course, 
understood that each and every module depicted in Fig. 
1 is available and can be programmed to render the 
accomplished task with a minimum of effort. 

Thus, the above described techniques are 

20 well within the ken of one skilled in the art, and the 
aspect of providing a desired video image within a 
recognizable pattern associated with a video display 
is readily achieved by these techniques „ It is, of 
course, understood that the pattern recognition 

25 algorithm has been basically described as related to 
sporting events or events where the field or other 
particular object is displayed fairly often during the 
televised presentation. 

It is, of course, understood that the 

30 techniques described herein will have applicability to 
other televised events such as contests, game shows 
and so on and is not limited in any manner to sporting 
events . 

i 
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Each and every module as indicated above has 
been further referenced in greater detail in the 

disclosed prior art- 
It is, therefore, indicated that the 
invention as described herein has great utility, and 
it is understood that alternate embodiments can be 
employed but tbe general content and scope of the 
invention is embodied in the enclosed claims. 
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Claims 

1. Apparatus for altering a video image 
display to provide a substituted display of desired 
indicia within a preselected portion of said video 
image display and independent of the size of said 

5 selected portion which size is strictly a function of 

the TV camera, said TV camera operable at various 
different perspectives employed to create said 
display , comprising: 

means for selecting said portion of said 

10 display to be substituted, 

means responsive to said video signal 
display for recognizing said selected portion of said 
display, 

means for generating video signals 
15 indicative -of —said desired -indicia- to be substituted, 

means responsive to said desired indicia 
signals and said video signal for inserting said 
indicia into said video image display at said 
preselected portion. 

2. The apparatus according to Claim 1, 

further including: 

audio processing means responsive to audio 
signals associated with said televised scene to alter 
5 said substituted display portion according to said 

audio signals, 

3 The apparatus according to Claim 2, 
wherein said display indicia is modulated according to 
the intensity of said audio signals, which signals 
exceed a given threshold. 

4. The apparatus according to Claim 1, 
wherein said means for selecting includes a light pen 
for outlining said portion of said display to be 
selected and means for storing said selected outline. 



BNSOOdD: <WO... .. ... ^8302324* 



WO 93/02524 



PCT/US91/05174 



-26- 

5. The apparatus according to claim 1, 
wherein said desired indicia includes an advertising 
message- 

6. The apparatus according to Claim 1, 

further including: 

delay means responsive to said recognized 
video signal and said video signals representative of 
5 said desired indicia for showing the same on a frame- 
to-frame basis to enable the insertion of said indicia 
on a real time basis. 

7. The apparatus according to claim l f 
wherein said means responsive to said video signal 
display includes pattern recognition means responsive 
to said selected portion of said display to provide 

S signals indicative of said selected portion 
independent of the size of said portion with respect 

to said display. 

8* The apparatus according to Claim 7, 
wherein said pattern recognition means includes means 
for performing a pyramid algorithm. 

9. The apparatus according to Claim l f 
further including means responsive to colors present 
in said video display to inhibit display substitution 
during the presence of selected colors , whereby 

5 replaceable pixels can be substituted for non- 
replaceable pixels. 

10. A method for altering a video image to 
provide a substituted display of desired indicia 
within a preselected portion of said video image 
display on a frame-to-frame basis and independent of 

5 the size of said selected portion on a frame- to- frame 
basis which size is a function of the TV camera 
perspective employed to create said display, 
comprising the steps of r 
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selecting said portion of said display to be 
10 substituted, 

recognizing said ©elected portion of said 
display on a frame- to- frame basis and independent of 
the size of said portion with respect to said display, 

generating a video image of said desired 

15 indicia, and 

inserting said image of said desired indicia 
within said recognized portion of said display on a 
frame-to- frame basis * 

11. The method according to Claim 10, 
wherein the step of generating a video image of 
desired indicia includes generating a video image of 
an advertising format such as a manufacturer's 
5 trademark or logo. 

12 •=^Phe = method^according, — r-^to — Claim -11, 

wherein said selected portion of said display to be 
substituted comprises the boundaries of a sports 
court, with said desired indicia including enhanced 
5 replicas of said boundaries. 

13. The method according to Claim 10, 
further including the step of 2 

detecting the intensity of audio signals 
emanating from said televised scene and altering the 
5 inserted image according to said detected audio. 

14- The method according to Claim 10, 
further indicates the step of: 

selectively responding to a color present in 
said display for inhibiting the insertion of said 
5 desired indicia within said selected portion according 
to the presence of said color. 

15. The method according to Claim 10, 
further including the step of: 
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storing said selected portion of said 
display in a memory prior to inserting the same into 
5 said stored selected portion. 

16 ♦ The method according, to Claim 10, 
wherein the step of recognizing includes , 

applying a pattern recognition 
algorithm to said video display after selecting said 
5 portion, wherein said algorithm is a pyramid algorithm 

capable of processing said displayed image enabling 
recognition of said selected portion independent of 
the size, location or orientation of the same with 
respect to said display on a frame-to-frame basis- 

17 « The method according to Claim 16 , 
wherein said algorithm is the Burt Pyramid Algorithm, 

18. The method according to Claim 10/ 
wherein the step of selecting includes outlining said 
selected portion of said display with a light pen and 

storing the outline of said light pen 
5 indicative of said selected portion* 

19. The method according to Claim 10/ 
further including the step of: 

modulating said inserted video image 
according to the sound intensity emanating from said 
5 televised scene. 

20. The method according to Claim 10, 

further including the step of: 

detecting motion in said display during a 
frame and inhibiting said video image of said desired 
5 indicia into said display according to said detected 

motion* 
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